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Study  Report 


Purpose 

The  purpose  of  this  study  report  is  to  discuss  NPRST’s  findings  and  conclusions 
during  the  verification  of  the  Center  for  Career  Development  (CCD)  Retention  Model. 

Background 

PERS-6  provides  a  monthly  executive  summary  of  enlisted  reenlistment  and  attrition 
rates  by  length  of  service  zone,  e.g.,  Zones  A  through  E  to  Senior  Navy  leadership  (CNO, 
CNP,  etc.).  To  understand  these  rates  a  simple  model  was  developed,  using  Microsoft 
Excel,  to  evaluate  the  effects  on  enlisted  end-strength  as  reenlistment  and  attrition  rates 
change.  Concentrating  on  these  rates,  the  model  has  a  total  of  13  inputs:  actual  begin 
strength  at  the  beginning  of  a  12-month  period,  number  of  planned  accessions  minus 
TAR  Enlistment  Program  (TEP)  during  a  12-month  period,  reenlistment  rates  generated 
from  the  Retention  Monitoring  System  (RMS)  by  zone  during  the  12-month  period, 
attrition  rates  by  zone  during  the  1 2-month  period,  and  a  predetermined  value  for 
USN/USNR  other  accessions.  The  model  has  only  ONE  output,  enlisted  end-strength  at 
the  end  of  a  12-month  period. 

Verification 

Regardless  of  the  problem  and  the  modeling  paradigm  that  is  used  to  model  a  system, 
a  modeling  team  amplifies  the  success  of  a  simulation  study  when  the  model  has  been 
verified,  validated,  and  accredited  for  use  to  analyze  a  specific  problem'.  This  process  is 
called  Verification,  Validation,  and  Accreditation,  or  VV&A,  and  is  usually  the 
responsibility  of  the  model’s  sponsor.  The  model’s  sponsor  is  the  agency  that  provides 
the  funding.  The  sponsor  is  closely  associated  with  the  ultimate  end-user  and  may  even 
be  the  end-user.  In  this  case,  CCD  is  acting  as  the  sponsor  and  the  user,  however,  results 
implicit  and  explicit  from  this  model  are  likely  to  be  used  by  a  wider  audience,  i.e., 
within  Navy  Personnel  Command. 

VV&A  terms  are  defined;  Verification  is  concerned  with  determining  whether  the 
conceptual  model  (model  assumptions)  have  been  correctly  translated  into  a  computer 
“program,”  i.e.,  debugging  the  computer  program.  Although  verification  is  a  simple 
concept,  debugging  a  model  can  be  a  difficult  and  arduous  task  due  to  the  potentially 
large  number  of  variables  and  logic  paths.  Validation  is  the  process  of  determining 
whether  a  model,  as  opposed  to  the  computer  program,  is  a  reasonable  representation  of 
the  system  being  studied,  for  the  particular  objective(s)  of  the  study.  Lastly,  a  model,  and 
its  results,  will  be  accredited  if  it  has  “creditability.”  A  model  is  credible  if  the  user 
accepts  it  as  “accurate.”  A  credible  model  is  not  necessarily  valid,  and  vise  versa.  Also,  a 
model  can  be  credible  and  NOT  actually  used  as  an  aid  in  making  decisions.  Although 


'  Myers,  R.  &  McDevitt,  M.  (in  press).  Analytical  Modeling  Methodologies  for  Personnel  Policy 
Development  and  Decision  Support. 
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our  analysis  of  the  model  will  help  to  support  the  model’s  accreditation,  our  analysis  is 
focused  on  verification  only. 

Mathematical  Model 

In  general,  the  Retention  Model  is  a  combination  of  a  mathematical  model,  and  a 
forecasting  technique.  For  this  particular  projection  model,  the  basic  mathematical  model 
is: 

Projected  End-strength  =  Begin  Inventory  +  Accessions  -  Losses  +  Error 

This  equation  describes  the  relationship  between  error,  losses,  accessions,  begin 
inventory,  and  projected  end-strength.  This  mathematical  relation  describes  the  operation 
of  determining  projected  end-strength  -  or  the  mathematical  model  of  projected  end- 
strength.  Frequently,  mathematical  models  describe  functional  relationships. 

Projected  End-strength  =  /(Begin  Inventory,  Accessions,  Losses) 

This  could  also  be  described  as  projected  end-strength  being  dependent  on  begin 
inventory,  accessions,  and  losses,  where  projected  end-strength  is  the  dependent  variable, 
and  begin  inventory,  accessions,  and  losses  are  independent  variables.  Throughout  this 
report  predicted,  projected,  and  forecasted  will  be  used  interchangeably. 

Forecasting  Technique  and  Error  Verification 

The  retention  model  generates  a  forecast  of  enlisted  end-strength  for  the  end  of  a  12- 
month  period.  The  forecasting  technique  used  is  known  as  extrapolation.  Extrapolation  is 
a  quantitative  forecasting  tool  that  uses  ONLY  the  previous  value  of  the  variable  being 
forecast  in  the  analysis.  Extrapolation  provides  a  convenient  way  to  generate  quick  and 
easy  forecasts  for  the  SHORT  time  horizon;  i.e.,  days,  weeks,  months,  quarter,  and  at 
most  a  year  ahead.  The  benefit  of  this  technique  is  that  it  generally  requires  little  to  no 
historical  data  prior  to  the  implementation  of  the  technique,  and  it  requires  only  simple 
arithmetic  (as  discussed  previously)  to  generate  a  forecast^. 

There  are  three  basics  assumptions  in  using  extrapolation:  forecasts  are  NOT  affected 
by  environment,  forecasts  will  NOT  immediately  recognize  turning  points,  and  trends  can 
be  identified  and  separated.  The  only  variables  used  in  the  extrapolation  are  the  variables 
being  forecast,  and  the  variables  used  in  the  equation.  Therefore,  all  other  influences  that 
may  impact  the  variable  are  excluded  from  the  analysis  and  the  forecast.  The  forecast  is 
only  based  on  previous  actual  values  of  the  variables.  Therefore,  the  forecast  will  not 
change  direction  until  after  the  actual  data  has  shown  that  change.  The  technique 
generally  expects  stationary  data  (no  trend)  and  if  a  trend  is  present  in  the  data,  will 
usually  treat  the  trend  as  a  straight  line  (linear)  trend. 

The  evaluation  of  any  forecasting  technique  relies  primarily  on  the  comparison  of  the 
forecasts  with  the  corresponding  actual  values.  Some  evaluation  methods  are^: 


^  Armstrong,  J.  (200 1 ).  Principles  of  Forecasting,  A  Handbook  for  Researchers  and  Practitioners.  Norwall, 
MA.  Kluwer  Academic  Publishers. 

^  Jensen,  A.  (n.d.).  Naive  Forecasting  Techniques.  Retrieved  August  7,  2003. 
htty://www.csus.edu/indi  v/j/jensena/mgml  1 05/naive_0 1  .ppt. 
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•  Mean  Error 

•  Mean  Absolute  Error 

•  Mean  Squared  Error 

•  Absolute  Error 

•  Percentage  Error 

The  Mean  Error  can  be  very  misleading.  A  Mean  Error  value  of  zero  can  indicate  that 
the  method  forecast  the  actual  values  perfectly  (unlikely)  or  that  the  positive  and  negative 
errors  cancelled  each  other.  Because  of  this  cancellation  effect,  it  tends  to  understate  the 
error.  The  Mean  Absolute  Error  is  a  way  of  dealing  with  the  understatement  of  Mean 
Error.  By  using  the  absolute  values  of  the  error,  the  mean  gives  a  better  indication  of  the 
model’s  fit.  The  Mean  Squared  Error  eliminates  the  positive/negative  problem  by 
squaring  the  errors.  The  result  tends  to  place  more  emphasis  on  the  larger  errors,  and, 
therefore,  gives  a  more  conservative  measure  than  the  Mean  Absolute  Error.  The 
previous  three  measures  are  “series  specific;”  i.e.,  they  only  allow  evaluation  of  the  series 
that  generated  the  errors.  The  Absolute  Error  is  the  absolute  value  difference  between  the 
forecast  value,  and  its  actual  value,  and  sometimes  is  taken  without  absolute  value.  When 
taken  without  the  absolute  value,  the  result  will  maintain  its  signed  value,  e.g.,  exact 
error.  The  Percentage  Error  is  100  percent  times  the  relative  error,  whereas  relative  error 
is  the  forecast  value  minus  the  actual  value  divided  by  the  actual  value.  The  Percentage 
Error  uses  the  exact  error  relative  to  the  actual  value,  and  is  designed  to  allow  comparison 
of  the  results  with  different  models. 

Since  we  are  concerned  with  the  accuracy  of  the  end-strength  projection,  based  on  10 
data  points,  we  will  use  Absolute  Error  and  Percentage  Error  to  measure  the  prediction 
accuracy  of  the  model. 

Absolute  Error  =  |  Predicted  Value  -  Actual  Value  \ 

Predicted  Value  -  Actual  Value 

Percentage  Error  = -  *100 

Actual  Value 

Equations 

Theoretical  Equation: 

Projected  End-strength  =  Begin  Inventory  +  Accessions  -  Losses  +  Error, 

Whereas, 

Begin  Inventory  =  onboard  enlisted  inventory  at  the  end  of  the  previous  FY  (or  at 
the  end  of  a  12-month  period) 

Accessions  =  planned  gains 

Losses  =  Attrition  losses  +  E AOS  losses 

Attrition  losses  =  (Begin  Inventory  +  Accessions)  *  Attrition  Rate 
EAOS  losses  =  (I  -  Reenlistment  Rate)  *  Eligibles 
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Eligible  <  90  days  prior  to  EAOS  and  Non~Eligible  >  90  days  prior  to 
EAOS 

Error  =  randomly  distributed  error  term,  which  represents  the  difference  between 
overestimation  and/or  underestimation  of  independent  variables 

Figure  1.  Describes  the  process  by  which  Non-EAOS  Inventory  and  Reenlistments 
are  derived. 


Apply  %  at  EAOS  ( 1  -Attrition)  Rate 


Figure  1.  Equation  Diagram. 


Eligibles  are  computed  by  applying  the  percentage  at  EAOS  (historical  average)  to 
the  current  inventory.  The  historical  average  is  determined  from  RMS  and  used  as  input. 

The  retention  model  uses  the  following  equations: 

Projected  Inventory  (end-strength)  =  Non  EAOS  Inventory  ^^eenlistments, 
Whereas, 
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Inventory  ~  Begin  Inventory  +  Accessions 

Accessions  include  Prior  service  (NAVET/OSVET,  Reserve  Recall),  and  Non-prior 
service  (USN  and  USNR),  taken  from  the  N13  accession  implementation  plan.  For  the 
purpose  of  attrition  calculation,  accession  attrition  is  calculated  using  the  same  attrition 
rates  as  non-eligible.  Also,  the  mathematical  model  includes  a  predetermined  number 
(3737)  to  account  for  USN/USNR  Other  accessions.  This  number  is  an  approximation  of 
USN/USNR  Other  accessions,  based  on  fiscal  year  (FY)  2003.  In  reality,  it  is  subject  to 
change  for  upcoming  years. 

An  adjustment  value  is  necessary  because  it  is  assumed  that  USNAJSNR  Other 
accessions  attrite  at  a  greater  rate  than  any  other  population  in  the  model.  To  account  for 
this  unknown,  the  most  recent  historical  data  for  a  minimum  of  six  months  is  reviewed  to 
determine  the  numerical  adjustment  required  to  reduce  the  numerical  error  of  the  model 
for  the  six  months  being  evaluated.  This  evaluation  is  performed  on  a  by-month  basis. 

The  model  developer  chose  six  months  as  the  data  window,  because  the  analysis  revealed 
a  consistent  delta  for  a  6-month  period  between  actual  end-strength  and  the  model  results. 
Beyond  six  months,  the  delta  began  to  decrease.  This  should  be  analyzed  separately  for 
each  update  to  the  model  to  determine  the  best  data  window  for  this  adjustment. 

However,  the  model  developer  warns  against  decreasing  the  window  below  six  months. 

To  calculate  the  adjustment  value,  use  the  results  from  previous  six  months  to 
determine  the  actual  errors  (difference  between  forecasted  and  actual)  for  each  individual 
month.  For  example,  if  the  target  forecast  month  is  October  of  a  given  year,  calculate  the 
actual  error  for  a  minimum  of  six  previous  months,  e.g.,  September,  August,  July,  June, 
May,  and  April.  Next,  determine  the  minimum  (Min)  and  maximum  (Max)  actual  error 
for  the  6-month  period.  Then  choose  an  adjustment  value  that  is  the  midpoint  between  the 
six-month  maximum  and  minimum.  The  following  is  used  to  calculate  the  adjustment 
value  for  the  6-month  period: 

Max  -  Adjustment  =  Min  +  Adjustment 

Max  =  Min+  2Adjustment 

Adjustment  =  (Max  -  Min)/2 

Inventory  is  adjusted  for  attrition  via  the  following  equation: 

Non- Attrition  Inventory  =  (1  -  Attrition  Rate)  *  Inventory 

Inventory  is  further  revised  to  account  for  reenlistment  eligibles  via  the  following 
equation: 

Non-EAOS  Inventory  =  Non-Attrition  Inventory  -  Eligible  (eligible  to  make  a 
reenlistment  decision) 

Reenlistments  =  Reenlistment  Rate  *  Eligible 

Additionally,  the  model  uses  an  adjustment  feature  for  attrition  rates  to  provide 
fidelity  by  zone.  In  other  words,  the  model  provided  good  answers  for  the  forecast  of 
total  end-strength,  at  the  end  of  the  1 2  month  forecast  period.  However  when  looking  at 
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changes  in  attrition  rate  by  zone,  improvements  in  length  of  service  (LOS)  profile  were 
necessary  to  reflect  actual  results  in  current  attrition.  For  example  in  FY02,  we  knew  the 
average  inventory  in  Zone  A  was  158,800  (for  the  12  month  period).  Therefore  this 
adjustment  feature  is  used  to  achieve  the  158,800  Zone  A  LOS  population.  Calculate  the 
attrition  adjustment  factor  for  each  particular  fiscal  year  by  using  the  following  equation; 
(projected  inventory  by  Zone  [model]  /  average  actual  inventory  by  zone  [RMS]).  This 
adjustment  feature  is  imbedded  in  the  model. 

A  similar  adjustment  is  used  for  reenlistment  rate.  The  adjustment  for  reenlistment 
rate  is  used  to  achieve  the  current  number  of  eligibles.  First,  use  the  model  to  calculate 
the  total  inventory  in  each  particular  zone  after  attrition  occurs.  Then  the  adjustment  is 
used  to  create  the  correct  number  of  eligibles  for  a  recent  fiscal  year.  For  example,  in 
FY02,  we  know  the  number  of  eligibles  in  FY02  was  24,500  for  Zone  A.  Therefore  the 
adjustment  for  reenlistments  considers  the  fraction  of  the  population  after  attrition  to 
reflect  24,500  eligible  for  reenlistment  in  Zone  A.  Calculate  the  reenlistment  adjustment 
rate  for  each  fiscal  year  by  using  the  following  equation:  (actual  eligible  population  by 
zone  [RMS]  /  projected  population  after  attrition  by  zone  [model]). 

RMS  data  is  generated  as  follows: 

Reenlistments  +  Long  Term  Extensions 

Reenlistment  Rate  =  - 

Reenlistments  +  Long  Term  Extensions  +  EAOS Losses 

Non-EAOS  Losses 

Attrition  Rate  — - 

Non  EAOS  Inventory 

Both  reenlistment  and  attrition  rates  are  cumulative  averages  of  the  actual 
reenlistment  and  attrition  behavior  over  the  current  12  months  taken  from  RMS.  Non- 
EAOS  are  the  personnel  not  eligible  for  reenlistment. 

Forecast  Verification  Resuits 

The  actual  enlisted  end-strength  numbers  for  the  previous  1 0  years  were  provided  and 
verified  by  N 132  (Director,  Enlisted  Strength  Planning). 

Table  1  contains  the  end-strength  forecast  verification  results  for  the  past  ten  years 
(FY93-FY02).  The  rows  represent  fiscal  years;  the  columns  represent  actual  end- 
strength,  forecast  end-strength  using  the  retention  model,  and  calculated  Absolute  Error 
and  Percentage  Error. 
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Table  1.  Analysis  of  Forecast  Results 


Fiscal 

Year 

Actual 

End-strength 

Forecast 

End-strength 

Absolute 

Error 

Percentage 

Error 

1993 

439,433 

446,375 

6942 

1.580 

1994 

402,626 

408,000 

5374 

1.335 

1995 

371,670 

374.620 

2950 

.794 

1996 

355,048 

350,595 

4453 

-1.254 

1997 

335,267 

335,443 

176 

.053 

1998 

323,120 

318,015 

5105 

-1.580 

1999 

315,178 

312,167 

3011 

-0.955 

2000 

315,471 

313,586 

1885 

-0.597 

2001 

319,601 

318,208 

1393 

-0.436 

2002 

324,351 

321,836 

2515 

-0.775 

We  were  not  given  a  target  Percentage  Error  or  Absolute  Error.  However,  in  past 
experience,  we  have  been  using  two  percent  error  as  a  target  for  forecast  that  are  within 
six  months.  During  past  literature  reviews,  we  were  unable  to  determine  an  industry 
standard  for  percentage  error.  It  appears  standards  are  set  based  on  the  objectives  of  the 
particular  study. 

The  forecast  end-strength  was  calculated  as  follows:  the  begin  inventory  was  taken 
from  previous  year’s  actual  end-strength,  the  planned  accessions  were  taken  from  the 
N 1 3  accession  implementation  plan  for  the  year  being  forecast,  the  percentage  at  EAOS 
and  the  cumulative  12  months’  reenlistment  and  attrition  rates  were  taken  from  RMS. 
The  predetermined  additional  USN/USNR  other  accessions  (3737)  were  assumed  for 
each  fiscal  year,  without  exception.  Further,  the  end-strength  calculations  follow  the 
equation  diagram  (Figure  1)  outlined  in  this  study  report,  including  adjustments. 

The  range  for  percentage  error  was  -1.580  to  1.580.  The  largest  percentage  error 
occurred  in  1993  and  1998,  relative  to  each  individual  year.  For  1993,  the  model  over 
projected  the  end-strength,  while  in  1 998,  the  model  under  projected  the  end-strength. 
These  distinctions  may  be  relevant  depending  on  the  fiscal  year  being  analyzed  in  other 
studies  or  planning  exercises.  The  smallest  error  occurred  in  1997,  where  the  model  over 
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projected  the  end-strength.  Fiscal  years  1994  and  1996  were  closer  to  the  larger 
percentage  error  with  values  of  1.335  and  -1.254  respectively.  Typically,  we  would 
expect  to  have  less  of  a  percentage  error  given  the  known  variables,  however,  in  this  10 
year  period,  the  model  developer  assumed  several  constant  values,  including  number  of 
TAR  Enlisted  Program  (TEPs),  the  adjustment  value,  and  the  number  of  USN/USNR 
Other  accessions  for  each  year. 

Figure  2  shows  the  actual  end-strength  vs.  forecasted  end-strength  for  1993  thru  2002. 


End  Strength  Projection 
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Figure  2.  End-strength  Projection. 


Figure  3  shows  the  absolute  error,  which  was  calculated  based  on  the  absolute  value 
difference  between  the  actual  end-strength  and  the  forecasted  end-strength  (see  “error 
verification”). 


Absolute  Error 


Fiscal  Year 


♦  Absolute  Bror 


Figure  3.  Absolute  Error. 

Figure  4  shows  the  percentage  error,  which  was  calculated  based  on  the  percentage 
difference  between  the  actual  end-strength  and  the  forecasted  end-strength  (see  “error 
verification”). 
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Percentage  Error 


♦  Percentage  Error 


Figure  4.  Percentage  Error 

Conclusions 

The  Retention  Model  contains  sufficient  mathematical  equations  and  business  rules  to 
perform  alternative  analysis  of  the  impact  on  enlisted  end-strength — as  accessions 
change,  and  more  importantly  as  reenlistment  rates  and  attrition  rates  change. 
Additionally,  the  projected  end-strength  derived  from  the  mathematical  model,  including 
adjustments,  could  be  used  as  an  estimate  of  enlisted  end-strength  for  a  12-month  period. 
The  forecasting  processes  and  equations  are  logical  and  traceable. 

This  model  does  NOT  capture  the  level  of  detail  that  is  modeled  in  the  existing 
strength  planning  and  community  management  models  and  should  NOT  be  considered  as 
a  replacement  for  either.  However,  this  model  does  represent  a  good  supplement  to 
existing  planning  methods  and  tools.  Properly  employed,  this  tool  can  provide  a 
reasonable  forecast  of  Navy  end-strength  that  results  from  changing  accessions, 
reenlistment  and/or  attrition  goals. 
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Appendix  A 


A-O 


Glossary 


Accessions 

Gain  to  Strength 

Attrition 

Non-EAOS  loss  to  strength 

CDR 

Commander 

CNO 

Chief  of  Naval  Operations 

CNP 

Chief  of  Naval  Personnel 

CNRC 

Commander,  Navy  Recruiting  Command 

EAOS 

Expiration  of  Active  Obligated  Service 

Eligible 

Eligible  to  make  a  re-enlistment  decision 

End-Strength 

Inventory  (strength)  at  the  end  of  a  period  of  time 

LCDR 

Lieutenant  Commander 

Long  Term  Extension 

>24  months  extension  of  contract 

LOS  Zones 

A  =  <6  years  of  service 

B  =  6-1 0  years  of  service 

C  =  10-14  years  of  service 

D  =  14-20  years  of  service 

E  =  >20  years  of  service 

NAVET 

Navy  Veteran 

OSVET 

Other  Service  Veteran 

Reenlistment 

Renewal  of  enlistment  contract 

Reserve  Recall 

Reserve  member  called  back  to  active  duty 

TAR 

Training  and  Administration  of  Reserves 

TEP 

TAR  Enlistment  Program 

USN 

United  States  Navy 

USNR 

United  States  Navy  Reserve 

A-l 


Distribution 


AIR  UNIVERSITY  LIBRARY 

AIRFORCE  ARMSTRONG  LABORATORY  (CODE  13) 

ARMY  MANAGEMENT  STAFF  COLLEGE  LIBRARY 
ARMY  RESEARCH  INSTITUTE  LIBRARY 
ARMY  WAR  COLLEGE  LIBRARY 
ASSISTANT  DEPUTY  CHIEF  OF  NAVAL  OPERATIONS 
ASN  (M  &  RA) 

CANADIAN  DEFENSE  LIAISON  STAFF 
CENTER  FOR  NAVAL  ANALYSES  LIBRARY 
CHIEF  OF  NAVAL  PERSONNEL 

COMMANDER  NAVY  PERSONNEL  COMMAND  (POOB,  PERS-4,  PERS-6,  PERS-8, 
PERS-9) 

NAWCTSD 

HEAD  MANPOWER  PERSONNEL  TRAINING  BRANCH  (N8I3) 

HUMAN  RESOURCES  DIRECTORATE  TECHNICAL  LIBRARY 
JOINT  FORCES  STAFF  COLLEGE  LIBRARY 
MARINE  CORPS  RESEARCH  CENTER 
MARINE  CORPS  UNIVERSITY  LIBRARIES 
NATIONAL  DEFENSE  UNIVERSITY  LIBRARY 

NAVAL  HEALTH  RESEARCH  CENTER  WILKINS  BIOMEDICAL  LIBRARY 

NAVAL  POSTGRADUATE  SCHOOL  DUDLEY  KNOX  LIBRARY 

NAVAL  RESEARCH  LABORATORY  RUTH  HOOKER  RESEARCH  LIBRARY 

NAVAL  WAR  COLLEGE  LIBRARY 

NAVY  MANPOWER  ANALYSIS  CENTER 

ONR  (CODE  342) 

OPNAV  (CODE  Nl,  NIB,  NIO,  N1 1,  N12,  NI3) 

PENTAGON  LIBRARY 

USAF  ACADEMY  LIBRARY 

US  COAST  GUARD  ACADEMY  LIBRARY 

US  MERCHANT  MARINE  ACADEMY  BLAND  LIBRARY 

US  MILITARY  ACADEMY  AT  WEST  POINT  LIBRARY 

US  NAVAL  ACADEMY  NIMITZ  LIBRARY 

DEFENSE  TECHNICAL  INFORMATION  CENTER 


